We construct intersecting D-brane configurations that encode the gauge groups and field content of dual N=4 supersymmetric gauge theories in three dimensions. The duality which exchanges the Coulomb and Higgs branches and the Fayet-Iliopoulos and mass parameters is derived from the SL(2,Z) symmetry of the type IIB string. Using the D-brane configurations we construct explicitly this mirror map between the dual theories and study the instanton corrections in the D-brane worldvolume theory via open string instantons. A general procedure to obtain mirror pairs is presented and illustrated. We encounter transitions among different field theories that correspond to smooth movements in the D-brane moduli space. We discuss the relation between the duality of the gauge theories and the level-rank duality of affine Lie algebras. Examples of other dual theories are presented and explained via T-duality and extremal transitions in type II string compactifications. Finally we discuss a second way to study instanton corrections in the gauge theory, by wrapping five-branes around six-cycles in M-theory compactified on a Calabi-Yau 4-fold.